record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it by localai-bot · Pull Request #1081 · mudler/vllm.cpp

localai-bot · 2026-08-16T23:27:36Z

The NemotronH model-matrix row said the architecture was BLOCKED on
KERNEL-SSM-MAMBA (#496)
because "the Mamba2 SSD core is unported". That kernel landed on 2026-08-13.
The row also carried ☐ required for its spec and none for its evidence while
five specs were committed and the paged forward, the device MoE arm and the ABI
driver had all landed. This change corrects the record. It does not move the
row's lifecycle state.

FOLLOWING_AGENTS_PROTOCOL

Why a directory listing hid the blocker

#496 landed its host arm at
47960a009 (W1) and its CUDA arm at 43a6c5518 (W2). The kernel is
src/vt/cuda/cuda_mamba2_ssd.cuh — a 692-line header, #included at
cuda_gdn.cu:48 and registered as kMamba2ChunkScan at cuda_gdn.cu:6669, not
a translation unit of its own. A src/vt/*mamba* FILE GLOB therefore returns
nothing and reads as absence, and that glob is the search both the row's text and
#1074 rested on.
nemotron_h.cpp:597 calls vt::Mamba2ChunkScan today, and nemotron_h.cpp:354
calls vt::MoeRelu2. #496 stays open for its GENERIC MambaSpec producer, which
this architecture does not need, because A1 made the runner read the model's own
KV spec.

Worth carrying: #496's own issue TITLE still asserts "Mamba2 SSD core is
unported", so the false claim has a second live copy this change cannot reach.

Three claims in the row were FALSE and are corrected: the KERNEL-SSM-MAMBA
block, the non-gated relu² MoE "exists nowhere locally", and ModelOpt
MIXED_PRECISION loading "exists nowhere locally". One was TRUE and stays: the
DeepSeek-style MTP head does not exist locally, and it remains owed as W5.

The row STAYS `INVENTORIED`

This is the part an earlier draft of this description got wrong, and the
correction is the point of the change. That draft moved the row INVENTORIED ->
PARTIAL with the mark 🚧, the rollup INVENTORIED 324->323 and PARTIAL
22->23, and the two prose counts 53->54. None of that is in this diff.

Measured on this head against origin/main:

the row's lifecycle cell is `INVENTORIED` on both sides;
the rollup table .agents/model-matrix.md:76-86 is BYTE-IDENTICAL to main's
— INVENTORIED 324, PARTIAL 22, ACTIVE 10, SPIKE 9, BLOCKED 5, DONE
3, READY 3, GATING 1, total 377;
scripts/check-agent-record.py derives MODEL=377 and its model inventory
{rows: 324, memberships: 373, architectures: 356, targets: 310, modules: 261}
unchanged.

The lifecycle move is left to whoever re-runs the gate against main. What
survives is the part that was never about a state change: the row's FALSE claims,
and its corrected code and test anchors.

The A3 gate PASSES, and it belongs to a named tree

An earlier draft recorded this gate as PENDING, because it read
#1221's pull request BODY as the
current state. That body's "Still owed: the sm_121a green-after re-run" was true
when written and stale by the time it was read. A body is a snapshot of its
author's knowledge at write time, not a live field.

The artefact is on the NAS at /usr/local/nas_share/rc/nh1157/. Verbatim from
gate_fixed.out:

TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
STRICT PASS

against the pinned oracle vllm=0.23.1rc1.dev1511+g555967922 on
nemotron-3.5-lightning-30b-nvfp4 at revision
29f2d1746d8f41e316523194b19018707749b1b1.

It is the DEVICE leg. The binary is libvllm 0.0.3+cuda, and cfg.log
records fp4-mma, cutlass-nvfp4 and cutlass-fp8 as ENABLED for [121a]. The
run logs Asynchronous scheduling is enabled (max_concurrent_batches=2), which
is precisely where device_token_ids is non-null; on the host queue it is always
null and the #1157 defect
cannot arise. The decisive evidence is neither of those: the same binary on the
same checkpoint with only nemotron_h_device.cpp reverted to the fix's
parent scores 4/24 (full rows=0, short rows=3) and bails at 8 generated tokens.
A host-leg run would have been unmoved by that revert. The delta is the proof;
the pass on its own is not.

The pass was measured on the tree that landed as 0ea5d249f, and there is no
run against current main. #1221 MERGED on 2026-08-18 as 0ea5d249f, which is
main's last touch of nemotron_h_device.cpp, so the repair the 96/96 depended
on is on main. That is not the same fact as a gated main: the 96/96 was taken
on that branch tree, main has advanced since, and nobody has re-run it. Every
row here therefore names the SHA the measurement belongs to and records no pass
on main. An earlier draft of this description instead called that branch "OPEN
and CONFLICTING" with "the merge is owed", and quoted a shipped cell as reading
"on #1221's OPEN branch, not on main". Neither is true here, and that quoted
text ships nowhere: grep -c "OPEN branch" over docs/ and
.agents/model-matrix.md is 0.

No throughput, latency or memory figure is claimed anywhere in this change.
gate_fixed.out carries 264.4s to load and 327-343s per 32-token prompt. That is
a correctness run whose lm_head and 46 FP8 mamba projections still execute
host-side. Both public rows say "NOT a benchmark" in those words. One config
caveat travels with the run: --gpu-memory-utilization 0.92 did not size the KV
pool, which fell back to 256 blocks
(#83).

The stale `## Owed` bullet for #1217

The ## Owed entry for #1217
in .agents/specs/nemotron-h-model.md was written while #1221 was open, and this
branch ADDED it: origin/main carries the phrase "open and unmerged" zero times.
It was false three ways at its own head, and the last commit repairs it.

the bullet said	this head has
#1221 "is open and unmerged"	merged 2026-08-18 as `0ea5d249f`
#1217 "is recorded NOWHERE in this tree"	`.agents/issue-index.md:403` and `nemotron-h-a2p-paged-forward.md:901`
"the standing reason this row's end-to-end gate is not a pass"	the same file's `## Now` records 96/96 `STRICT PASS`

The "recorded nowhere" claim WAS true at b626be75a, which is verifiable —
git show b626be75a:.agents/issue-index.md | grep -c 1217 is 0. The merge
8a00afcbe brought both records in. Both anchors were re-derived at this head:
issue-index.md:403 carries the row under the owning row
MODEL-NEMOTRON-H-ABI-A2P, and nemotron-h-a2p-paged-forward.md:901 is the
first bullet under that spec's ## 11. Owed.

The duplicate-key argument for withholding an index row is kept as history and
marked expired. Appending a row now would create the very duplicate that argument
was written to avoid, because the row is already on main. Nothing is owed to
this reconcile for #1217. The issue stays open as a seam defect owned by
MODEL-NEMOTRON-H-ABI-A2P.

Two defects found and fixed in flow

docs/BENCHMARKS.md carried the key NemotronH paged forward twice on this
branch and once on main, because the branch APPENDED where it should have
UPDATED, and the two rows contradicted each other (4/24 with a re-run pending,
versus 96/96). They are collapsed to one row on the key. grep -c for that key
is now 1 here and 1 on main. Nothing gates this:
scripts/check-public-doc-tables.py holds the page's size and shape, not its key
uniqueness. The row's position moved when the duplicate was collapsed and it now
follows Record-anchor ratchet; the key and content are correct, and reordering a
file that other pull requests edit was not worth the conflict.

docs/STATUS.md was restructured under this branch by
#1491, which reorganised it by
surface and deleted the per-model table, so the merge conflicted for real. The
page is resolved to main's whole and the record is re-expressed as a one-line
in-place edit to the Text generation row.

Which projections this change owes, and why

scripts/check-doc-checkpoint.py was instrumented on this exact diff rather than
reasoned about. classify() returns classes {'feature_surface'} and
lifecycle_moves []. Its STATES tuple (:56-65) contains neither
INVENTORIED nor PARTIAL, so it is structurally blind to this row's state and
demands nothing for it. MEASUREMENT_RECORDS is ('.agents/benchmark-record.md',)
and this change does not touch that file, so the measurement class is not
triggered either.

docs/STATUS.md, docs/BENCHMARKS.md and the spec's ## Now are therefore
written because AGENTS.md ## Public documents owes them, not because a gate
asked. docs/FEATURES.md IS touched, at :148: the NemotronHForCausalLM row
said nothing about the end-to-end token gate before, and its correctness column
now reads "A3 e2e gate 96/96 STRICT PASS on GB10 at 0ea5d249f (#1221); NO run
against current main" — the two facts together, including the one that keeps the
claim honest.

The docs/STATUS.md edit takes oversized_cells from 2 to 3 against a ratchet
of 44, and long_paragraphs stays 0 against 75. The page is inside its ratchet
with room; no ratchet is raised.

`.agents/issue-index.md`

GitHub does not run the merge=union driver .gitattributes:7 sets, so this
branch has repeatedly discarded the union auto-merge, taken main's file whole,
and re-appended only the rows whose KEY main lacks: #1074 and #1080. main
already carries a #1068 row, so that one is not re-appended.

Measured on this head against origin/main: 511 rows, of which main's 509
are all present, in order, byte-identical; 0 removed; exactly 2 added.
scripts/check-issue-index-append-only.py exits 0. The file is no longer a
strict BYTE prefix of main's, because main's newest rows now follow this
branch's two, and the append-only rule is about rows rather than bytes.

#1080 is re-measured rather than carried: after this change repairs
nemotron-h-model.md's ## 7. Now to ## Now, 15 specs still write the
numbered spelling, up from the twelve #1080 was filed against. A growing
population is the argument for the checker-semantics fix over a rename sweep, and
both need their own spec and a red-before test. It is listed under ## Owed in
nemotron-h-model.md.

Gates

Run on 9ab30316759c0684da9b48cfc8a262b13c757b5f, this pull request's head, with
origin/main at 947e5f648552192d08cad4e636953be57bf34ffb.

scripts/agent-preflight.sh --fail-on-skip reports ZERO gates SKIPPED. That
is falsified rather than read off a green line. RANGE_COUNT is 24, and both
trailer gates were proven to READ that range by mutation: a detached scratch
commit with no trailer block makes check-commit-trailers.py exit 1 naming
e54386c0e05d, and a scratch commit whose subject ends in a period makes
check-commit-style.py exit 1 naming e7105f9fae0c. The tree was restored to
9ab303167 after each, verified by SHA and a clean git status.

Bare, on the same head, each exit 0:

check-doc-checkpoint.py --base 947e5f648 --head HEAD   OK: public documents match the claims this change makes.
check-public-doc-tables.py                             OK
check-supported-models.py                              OK: exactly the 40 architectures registered
check-agent-record.py                                  agent record OK: ENGINE=169 MODEL=377 QUANT=84 KERNEL=57 BACKEND=85 ANCHOR-ROT=37
check-issue-index-append-only.py                       OK: issue index append-only
check-symbol-anchors.py                                in-repo checked 255 (fresh 255, stale 0); buckets sum 829 vs 829; floor 85
check-now-current.py                                   OK

scripts/agent-preflight.sh exits 1 on test_cpu_x86_llamacpp_floor ALONE. That
is #618 and not a finding: the
box read load=74.80 and the harness reported
waiting for quiet: 15s busy=114% builders=0 load=71.21. The
test reads scripts/cpu-x86-llamacpp-floor.sh and
docs/bench-evidence/cpu-x86-llamacpp-20260811.md, and this change touches
neither — it touches six record files and zero code files.

Inherited, not introduced

windows-msvc-cpu and windows-msvc-vulkan fail on every pull request and have
no main baseline (#584,
#968).
test_cpu_x86_llamacpp_floor exits NO_QUIET_WINDOW under box load
(#618). Five CPU suites are
standing red on main under
#1464, separately owned; this
change has zero code files and cannot reach them.

Closes #1074

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…o, and a file glob is why nobody saw it The NemotronH model-matrix row described `main` as of 2026-08-12 and had not moved since. It read `INVENTORIED`, blocked on `KERNEL-SSM-MAMBA` (#496), and named three things that "exist nowhere locally". Every one of those claims was re-checked against `main` at `10002648199cfbbaf1e423f7c80cacb2f4b56366` rather than inherited, including the ones the filing issue asserted. Two of the three now exist. The non-gated relu2 MoE landed at `4d0c399e1`: `vt::MoeRelu2`, called from `nemotron_h.cpp:354`. ModelOpt `MIXED_PRECISION` per-module loading landed at `1bc5ef82c`. The third, the MTP head, is still genuinely owed as W5, and the loader defers its 270 tensors by name. The blocker itself is the finding. #496 W1 landed the three Mamba2 host references at `47960a009` and W2 landed the CUDA arm at `43a6c5518`. The kernel is `src/vt/cuda/cuda_mamba2_ssd.cuh`, a header included by `cuda_gdn.cu` rather than a translation unit of its own, so a `src/vt/*mamba*` file glob finds nothing and reads as absence. That is exactly the search this row's text and the filing issue both rested on. NemotronH calls those ops at `nemotron_h.cpp:597,620` today. #496 stays open for a GENERIC `MambaSpec` producer, which this model does not need, because A1 made the runner read the model's own KV spec. The row moves `INVENTORIED` to `PARTIAL`, with the rollup, the checklist entry and the projections that move owes. It does not move to `ACTIVE`, and the reason is a rule rather than a judgement: `check-agent-record.py` requires an `ACTIVE` row to name a `CLAIM-*` row that a claim source carries, no claim record claims this row, and authoring one for another session's in-flight work would be a fabricated record. `PARTIAL` is what the existing evidence backs, and it is what the two Qwen3.5 text-only arms carry for the same posture. Nothing here claims reachability. The forward reached through `ModelRegistry::Forward` is the host reference, so `nemotron_h_registry.cpp:162` still refuses paged and batched decode by name. There is no `examples/nemotron_h_gen`, no ABI token gate, and no throughput, latency or memory number, and `docs/BENCHMARKS.md` records that as a gap rather than a number. Found while doing this: `check-doc-checkpoint.py` matches a spec's live-position section with `^##\s+Now\s*$`, and thirteen specs write it as `## N. Now`, so a lifecycle move reds on a section that is present. Filed as #1080. This spec is the thirteenth and is repaired here, because this change is what makes it the spec a moving row links. The other twelve are listed under `## Owed`. Closes #1074 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…-- issue-index reconciled BY HAND `origin/main` moved while this reconcile was written. The union driver merged `.agents/issue-index.md` by interleaving: it put this branch's two rows before main's `#1066` row, so main's file stopped being a prefix of ours. The index is an append-only log and the prefix property is what makes it one, so the auto-merge was discarded and main's file was taken whole, with this branch's two rows re-appended at the end. Verified: main's blob is a byte prefix of the result, 290 rows, no duplicate issue number. Nothing else conflicted. `docs/BENCHMARKS.md`, `docs/FEATURES.md` and `docs/STATUS.md` were touched on both sides in different places, and the diff against `origin/main` after the merge is exactly this branch's additions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` advanced and GitHub reported CONFLICTING on `.agents/issue-index.md`. GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so the resolution happens locally where it does. The union auto-merge of the index was DISCARDED rather than trusted: main's file taken wholesale, only this row's two rows re-appended, main's file asserted a strict PREFIX of the result. 293 rows, zero duplicate keys. `.agents/model-matrix.md` is a KEYED table, not an append-only log, and a union merge DUPLICATES a keyed row rather than merging it -- that has happened in this tree before. Verified it was NOT auto-merged (only issue-index.md, docs/BENCHMARKS.md and docs/FEATURES.md were), and that the NemotronH key appears exactly ONCE as a table row (`:286`), its other occurrence being the separate checklist table at `:145`. `check-agent-record.py` and `check-model-checklist.py` both exit 0. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` advanced (#1084 and #1085 landed) and this branch conflicted for two different reasons, resolved two different ways. `.agents/issue-index.md` is append-only under `merge=union`, which GitHub does not run -- so the union auto-merge was DISCARDED, main's file taken wholesale, and only this row's own rows re-appended. Main's file is a strict PREFIX of the result, zero duplicate keys. `docs/FEATURES.md` was a GENUINE content conflict, not a merge-driver artifact: both sides added a `NemotronHForCausalLM` row -- this branch's, and main's from #1085. A keyed table gets ONE row per key carrying BOTH sides' content, so the two were merged rather than one being chosen: - from this branch: the forward computes on landed `vt::Mamba2*` ops (#496) -- the finding that retired this row's false blocker -- and the loader's quantization breakdown (5935 NVFP4 g16, 46 FP8 W8A8, bf16); - from #1085: the A2-Q2a device MoE arm (23 blocks, NVFP4 g16 Marlin) and the sharper unreachability statement, G-SAFE refuses FIRST with A2-P owning the wiring; - from both: no e2e gate and no number. The merged row is 6 parts with a maximum cell of 214 against the 220 cap, so it fits without deleting anyone else's entry -- the "cap the entry, never the file" rule doing its job. `check-public-doc-tables.py` and `check-agent-record.py` both exit 0, and the tree carries no conflict markers. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` advanced (#1082 A2-P spec landed) and GitHub reported CONFLICTING on `.agents/issue-index.md`. GitHub does not run the `merge=union` driver that `.gitattributes:7` sets, so the resolution happens locally where it does. The union auto-merge was DISCARDED: main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted to be a strict PREFIX of the result. Zero duplicate keys. The earlier `docs/FEATURES.md` conflict on this branch -- both sides adding a `NemotronHForCausalLM` row -- stays resolved as one merged row per key, carrying the Mamba2-ops finding and the loader breakdown from this branch and the A2-Q2a device arm and G-SAFE wording from #1085. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…s row, not merged into it `main` moved 54 commits ahead of this branch and its NemotronH row is NEWER than this branch's: A2-R `598226e96`, A2-P `a6df72777`, A2-Q2a and the A3 driver `c83b96934` all appended to the same cell after this branch was cut. Merging the two texts would have carried this branch's "the step still REFUSES paged decode because the reached forward is the HOST reference" over the top of a paged forward that has since landed. So no keyed record was auto-merged. `.agents/model-matrix.md`, `docs/FEATURES.md`, `docs/BENCHMARKS.md`, `docs/STATUS.md` and `.agents/specs/nemotron-h-model.md` were taken from `b626be75a` WHOLE -- the tree was asserted byte-identical to main before any edit -- and the scoped edit was then written again against what main actually says. `.agents/issue-index.md` is append-only under `merge=union`, which GitHub does not run, so the union auto-merge was discarded, main's file taken whole, and only the rows whose KEY main lacks re-appended: #1074 and #1080. Main already carries a #1068 row, so that one is NOT re-appended. Main's blob is a strict BYTE PREFIX of the result (341815 of 344811 bytes), 366 rows, zero duplicate keys. What survives the rewrite is what is still true of main: the row reads `INVENTORIED`, its `Spike/spec` cell reads `☐ required` against five committed specs, its evidence cell reads `none`, and it still says the row is BLOCKED on `KERNEL-SSM-MAMBA` (#496) because "the Mamba2 SSD core is unported". That last claim is false and re-measured here. What does NOT survive: this branch's `docs/BENCHMARKS.md` row said the reached forward is the host reference and named A2-P as owed. Both are stale, and main already carries a correct `MODEL-NEMOTRON-H-ABI-A2P` row saying the A3 gate is PENDING. The replacement row is keyed to the MODEL row instead and points at that one rather than restating it. This branch's `docs/FEATURES.md` edit is dropped entirely: main's row already describes A2-P, A2-Q2a and G-SAFE correctly, and no feature surface moves in this change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…, and it was re-appended by hand `origin/main` advanced two commits (#1216, #1237) while this reconcile was being written. Neither touches `.agents/model-matrix.md`, `docs/STATUS.md`, `docs/BENCHMARKS.md` or `.agents/specs/nemotron-h-model.md`, so the scoped edit did not have to be rewritten a second time. `.agents/issue-index.md` did overlap: main gained three rows and the union driver interleaved them, putting this branch's two rows before main's new ones and destroying the prefix property that makes the file an append-only log. GitHub does not run that driver anyway. So the auto-merge was DISCARDED, main's file taken whole, and only the rows whose KEY main lacks re-appended: #1074 and #1080. Asserted: main's blob is a strict BYTE prefix of the result (346246 of 349242 bytes), 369 rows, zero duplicate keys. The delta against `cdfade6ab` is byte-for-byte the same five files it was against `b626be75a`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…sted `origin/main` advanced again (#1248) while this branch was being pushed, and this time it DID touch three of the same files: `.agents/model-matrix.md`, `docs/BENCHMARKS.md` and `docs/STATUS.md`. Git auto-merged all three, and AGENTS.md `## Records` says never to accept that for a keyed record. So the auto-merge was checked rather than trusted. `git diff` against `10fe7f475` returns 13 changed lines in `.agents/model-matrix.md` and exactly one each in `docs/BENCHMARKS.md` and `docs/STATUS.md`, and every one of them is this branch's own scoped edit: the NemotronH row, the rollup, the two prose counts, the checklist entry, the open-gap row and the status clause. #1248's `MODEL-SPEC-deepseek-v4-dspark-...` row at `:546`, its DSpark benchmark row and its DSpark status paragraph are byte-for-byte equal to main's. No unrelated key moved. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…e fourth time `origin/main` advanced twice more (#1245, #1186). Only `.agents/issue-index.md` overlaps, and it overlaps every time because it is the one file in this change that every other branch also appends to. That is why GitHub keeps reporting this pull request CONFLICTING: it does not run the `merge=union` driver `.gitattributes:7` sets, so the conflict is real on the forge and absent locally. Resolved the only way that keeps the file an append-only log: the auto-merge was DISCARDED, main's file taken whole, and only the rows whose KEY main lacks re-appended -- #1074 and #1080. Asserted: main's blob is a strict BYTE prefix of the result (346713 of 349709 bytes), 370 rows, zero duplicate keys. Nothing else overlapped. The delta against `ae581da3e` is the same five files it was against `10fe7f475`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…DING because I read a body written before its own job ran The previous commit here wrote the A3 end-to-end token gate as `PENDING` and said no GB10 96/96 existed. That was wrong, and the way it was wrong is worth keeping: I searched `origin/main` and five `row/MODEL-NEMOTRON-H-*` branches, found nothing, and then took #1221's PULL REQUEST BODY as the current state. Its "Still owed: the sm_121a green-after re-run" was TRUE WHEN WRITTEN and stale by the time I read it. A body is a snapshot of its author's knowledge at write time, not a live field, and no amount of branch searching corrects for that, because the artifact was never in git at all. It is on the NAS. `/usr/local/nas_share/rc/nh1157/gate_fixed.out`: TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode) STRICT PASS against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4` at revision `29f2d1746d8f41e316523194b19018707749b1b1`. Verified by reading the files, not by trusting the report of them. It is the DEVICE leg. The binary is `libvllm 0.0.3+cuda` and `cfg.log` records `fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` `ENABLED for [121a]`. The run logs `Asynchronous scheduling is enabled (max_concurrent_batches=2)`, which is exactly where `device_token_ids` is non-null; on the host queue it is always null and the #1157 defect cannot arise. The decisive one is neither: the same binary on the same checkpoint with ONLY `nemotron_h_device.cpp` reverted to the fix's parent scores `4/24 (full rows=0, short rows=3)` and bails at 8 tokens. A host-leg run would have been unmoved by that revert. The delta is the proof; the pass alone is not. So the row now says the gate PASSES and says whose it is. It belongs to #1221, branch `row/MODEL-NEMOTRON-H-ABI-A2P-1157-fix` at `6e9e8955`, OPEN and `CONFLICTING`. `main`'s last touch of `nemotron_h_device.cpp` is `a6df72777` (A2-P), so `main` does not carry the fix and is not gated. A gate that passed and a tree that is gated are different facts, and this record keeps them apart. No performance number is claimed anywhere. `gate_fixed.out` carries 264.4s to load and 327-343s per 32-token prompt; that is a correctness run whose `lm_head` and 46 FP8 mamba projections still execute host-side. Both public rows say so in the words "NOT a benchmark", so nothing can later lift them into a grid. One config caveat also travels: `--gpu-memory-utilization 0.92` did not size the KV pool, which fell back to 256 blocks (#83). The row stays `PARTIAL`. `ACTIVE` still fails the `CLAIM-*` rule, and a gate that passes off `main` is not a reason to move a lifecycle state. `docs/FEATURES.md` moves in the same commit, because it has to. This commit edits `.agents/model-matrix.md`, which `scripts/check-doc-checkpoint.py` reads as a `feature_surface` change by path (`FEATURE_SURFACE_FILES`), and that class owes `docs/FEATURES.md` in the SAME commit -- the checker walks a range one commit at a time, so no later commit can pay the debt. The row for `NemotronHForCausalLM` said nothing at all about the end-to-end token gate while the matrix said it was PENDING, and it would have kept saying nothing now that the gate passes. It now carries the same two facts the matrix does, in the correctness column and in that column's own words: 96/96 `STRICT PASS` on GB10, on #1221's open branch and not on `main`. The added text is 90 characters, which leaves the cell at 192 of its 220-character cap and the row at 499 of 600, so no other row pays for it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

`main` gained #1221 while this branch was open, so `docs/BENCHMARKS.md` and `docs/STATUS.md` conflicted on the NemotronH rows. Resolved by taking main's rows as the base and correcting the one claim that main is now STALE on. Main says "GB10 read 4/24 ... sm_121a re-run pending a lease" and "sm_121a re-run pending". That was true when #1221 was written. The re-run has since happened: the A3 gate PASSES on GB10 at 96/96, `STRICT PASS`, mode=decode, against `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4` revision `29f2d1746d8f41e316523194b19018707749b1b1`, with a red-before of 4/24 on the SAME binary and checkpoint with only `nemotron_h_device.cpp` reverted. Artefacts: `/usr/local/nas_share/rc/nh1157/{gate_fixed,gate_red,cfg}.out`. `cfg.log` shows `fp4-mma`, `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and `fa2` all `ENABLED for [121a]`, so the build was not degraded, and both legs log `Asynchronous scheduling is enabled`, which is the path where `device_token_ids` is live and the defect lived. The delta, not the pass, is the proof: on the host leg reverting the fix would have changed nothing. STILL NOT A BENCHMARK. The 264.4 s load and 43 405 MB peak are recorded as a correctness run and labelled as such, because NVFP4 `lm_head` (A2-Q2b) and the 46 FP8 W8A8 mamba projections (A2-Q1, #940) still execute host-side. No throughput, latency or memory ratio is claimed on any axis. The index was reconciled by discard-and-reapply: main's file taken wholesale, only rows whose key main lacks re-appended, main's file a strict PREFIX. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…tted `main` is green again (#1297 fixed the gcc-16 `getpid` red; the qwen `build-test-cpu` red had already been fixed by `281b4bc76`), so this branch is merged forward to drop the stale inherited failures. `docs/BENCHMARKS.md` was a genuine content conflict: this branch and main inserted different rows at the same point. BOTH are kept, main's first. No row dropped, no ratchet raised. ANCHOR REPAIR. The merge rotted one citation and `check-agent-record.py` caught it: `stale' 33 > baseline 32`, naming `nemotron_h_device.cpp:1412 expected NemotronHPagedForward`. Main's count is exactly 32 with RC=0, and diffing the two report lists isolated the single extra entry as this row's, so it was repaired rather than the baseline raised. The symbol now lives at `:1474`. Worth recording because it nearly shipped: the citation is a markdown link, and the DISPLAY TEXT and the LINK TARGET are two separate spellings of the same line number. Repairing only `nemotron_h_device.cpp:1412` left `[nemotron_h_device.cpp:1474](...cpp#L1412)` -- a citation that READ correct and POINTED wrong, and which a reader checking the rendered text would have called fresh. Both halves are now `1474`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

Catch-up merge so the branch's gates run against the tree the records will land on. `git merge-tree --write-tree` reported no conflict. `.agents/issue-index.md` is the one keyed record both sides touch, and it is verified by key rather than trusted to the automatic merge: `origin/main` holds 451 rows, this tree holds 453, all 451 of main's rows are present in main's own order, and the two extra rows are this branch's own `#1074` and `#1080` entries. No key is duplicated. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…ce was reddening this branch (#1074) FOLLOWING_AGENTS_PROTOCOL `build-test-cpu` and both `sanitize-cpu` jobs were red here on one test, `175 - test_nemotron_h_paged_forward`, and the defect was never in this branch, which carries no product code at all. #1371 left `CPU_ATTN` as a name in a candidate list with no registrar, so the CPU inherited FlashAttention's `head_size % 8` constraint with no fallback. `9ecaf1bb3` (#1392) fixes that, and `f07f96e1c` (#1444) is the current `origin/main` that carries it. Merging the older SHA would have left this branch behind again, and `commit-trailers` and `commit-style` SKIP on a branch that is behind. Four files are touched by both sides, and each was verified rather than trusted to the automatic merge. `.agents/issue-index.md` is the keyed record: main carries 462 rows and this tree carries 464, all 462 of main's rows are present byte-identical and in main's own order, the preamble is byte-identical, and the two added rows are this branch's own `#1074` and `#1080`. Nothing is removed and no key is duplicated. `docs/BENCHMARKS.md`, `docs/FEATURES.md` and `docs/STATUS.md` are projections whose two sides edit different rows, and that is measured rather than asserted: the added and removed content lines of `merged vs main` hash equal to this branch's own delta, and those of `merged vs branch` hash equal to main's own delta, for all three files. No file is touched beyond what the merge itself requires. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…nion-append as a conflict (#1074) FOLLOWING_AGENTS_PROTOCOL This branch went `CONFLICTING`/`DIRTY` the moment #1409 merged, because both append a row to `.agents/issue-index.md` and GitHub does not apply the `merge=union` driver that `.gitattributes` declares for that path. It is the known forge artifact rather than a real conflict, and that was established before the merge rather than assumed: `git merge-tree --write-tree` over `origin/main` and this branch's tip exits 0 and writes tree `2404d135933ba312a147157ccab9bf315502567b`, with no conflicted path. The resolution is to re-merge, not to debug the forge. Two keyed records are touched by both sides, and each was verified by key rather than trusted to the automatic merge. `.agents/issue-index.md` stays a pure append: main carries 471 rows and this tree carries 473, all 471 of main's rows are present byte-identical and in main's own order, the preamble is byte-identical, the two added rows are this branch's own `#1074` and `#1080`, nothing is removed and no key is duplicated. `.agents/model-matrix.md` is the one that needed care, and it is NEW to this merge -- main did not touch it when this branch was last merged. It holds 373 keyed rows on both sides, 0 missing and 0 added and 0 duplicated, and exactly ONE row differs from main: `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm`, which is the row this reconcile exists to move. The file also stores DERIVED COUNTS beside its rows, which a textual merge cannot reason about, so they were checked rather than accepted: this branch moves `53 -> 54` engaged, `324 -> 323` `INVENTORIED` and `22 -> 23` `PARTIAL`, and main's own edits to the two `Qwen3_5*` rows touch ZERO count lines and change no lifecycle state -- they renumber `file:line` anchors inside rows that stay `PARTIAL`. The merged counts are therefore still the counts of the merged rows. `docs/BENCHMARKS.md`, `docs/FEATURES.md` and `docs/STATUS.md` are projections whose two sides edit different rows. Measured rather than asserted, for all three and for the model matrix: the added and removed content lines of `merged vs main` hash equal to this branch's own delta, and those of `merged vs branch` hash equal to main's own delta. No file is touched beyond what the merge itself requires. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…o its model row is resolved to main's page (#1074) FOLLOWING_AGENTS_PROTOCOL `docs/STATUS.md` is a REAL conflict this time, not the `merge=union` forge artifact the previous two merges were. #1491 (`aee6c48d6`) cut the page from 2906 lines and 65 table rows to 98 and 27, reorganising it BY SURFACE rather than by model, and the per-model table this branch edits no longer exists. The row the branch changes has no counterpart to merge into. The resolution is main's page, taken whole and byte for byte, so this commit carries no trace of the old structure. Re-expressing what that row recorded is a content decision and is deliberately NOT in this commit; it is the next one, so a reviewer can read the merge and the decision separately. The other keyed records merged clean and were verified by key rather than trusted to the automatic merge. `.agents/issue-index.md` stays a pure append: main carries 502 rows and this tree 504, all 502 of main's rows byte-identical and in main's own order, the preamble byte-identical, the two added rows this branch's own `#1074` and `#1080`, nothing removed and no key duplicated. `.agents/model-matrix.md` holds 373 keyed rows on both sides with 0 missing, 0 added and 0 duplicated. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…ncile rested on expired (#1074) FOLLOWING_AGENTS_PROTOCOL This reconcile was written while [#1221](#1221) was OPEN, and it moved the row `INVENTORIED` -> `PARTIAL` on the strength of an A3 gate that read 96/96 `STRICT PASS` on GB10. #1221 MERGED on 2026-08-18 as `0ea5d249f`, which is now `main`'s last touch of `nemotron_h_device.cpp`, so every "blocked on #1221 landing" sentence in this branch became false. What is recorded now is narrower than what the branch originally claimed, and deliberately so. The A3 gate read 96/96 `STRICT PASS` (mode=decode) on GB10, MEASURED ON THE TREE THAT LANDED AS `0ea5d249f`. That fix is on `main`. **No run against current `main` exists**, because `main` has advanced many commits past the measured tree. Naming the SHA a measurement belongs to is the whole point: an evidence line that names a tree it was not measured on has cost this repository before, and this change does not add another. **The row therefore STAYS `INVENTORIED`.** The lifecycle move is left to whoever re-runs the gate against `main`, and the derived counts in `.agents/model-matrix.md` are restored to main's `53`/`324`/`22` with the engaged-architecture row withdrawn. Measured rather than asserted: the row-state histogram over all 373 keyed rows is byte-identical to main's, so this change moves NO row's lifecycle state, and the count block is byte-identical to main's. What survives from the original reconcile is the part that was never about #1221: the row's `KERNEL-SSM-MAMBA` block was FALSE, and its corrected code and test anchors stay, so a later state move has its contract already written. `docs/BENCHMARKS.md` carried the key `NemotronH paged forward` (`MODEL-NEMOTRON-H-ABI-A2P`) TWICE on this branch and once on main, because the branch APPENDED where it should have UPDATED. The two rows contradicted each other: main's said GB10 read 4/24 with an sm_121a re-run pending a lease, and the branch's said GB10 read 96/96. The lease and the re-run happened, so main's row is stale on its face. They are collapsed to ONE row on the key byte-identical to main's, carrying the current projection. Nothing gates this: `check-public-doc-tables.py` holds the page's size and shape and not its key uniqueness, so it would have landed silently. `docs/STATUS.md` gets a one-line in-place edit to the `Text generation` row's "Open gate or limitation" cell, which is where #1491 routed this after it reorganised the page by surface. That is the shape `af25bd251`, `5702d8f83` and `d995c52f0` established for `Speculative decoding` and the diffusion row, and it satisfies the `lifecycle` obligation without growing a shrink-only page: 1 insertion, 1 deletion, 98 lines and 27 table rows before and after. Model-level detail is NOT reintroduced; it stays in [`docs/models/nemotron-3-5-lightning.md`](../docs/models/nemotron-3-5-lightning.md), whose "no token gate result exists" line remains true of `main`. The obligation still fires with no lifecycle move because `check-doc-checkpoint.py` adds the `lifecycle` class for a measurement change as well as a state change, so `STATUS` and `BENCHMARKS` are both owed and both written. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

… open, and this branch added that text (#1074) The `## Owed` entry for #1217 was written while #1221 was open and it kept three claims that are false at this head. It called #1221 "open and unmerged", although #1221 merged on 2026-08-18 as `0ea5d249f`, which the previous commit's own subject says. It said #1217 "is recorded NOWHERE in this tree", although the merge `8a00afcbe` brought both records in: `.agents/issue-index.md:403` carries the index row under the owning row `MODEL-NEMOTRON-H-ABI-A2P`, and `nemotron-h-a2p-paged-forward.md:901` lists the issue under that spec's `## 11. Owed`. That claim was true at `b626be75a` and is not true here. And it named #1217 "the standing reason this row's end-to-end gate is not a pass", which contradicts `## Now` in the same file, where the A3 gate reads 96/96 `STRICT PASS`. `git diff 6b48edb 414a800` shows this branch ADDED the phrase, and `origin/main` carries it zero times, so the stale text is this pull request's to remove rather than an inherited defect. The bullet now records where the record landed and cites both anchors, which were re-derived at this head. The duplicate-key argument for withholding an index row is kept as history and marked expired: appending a row now would create the very duplicate that argument was written to avoid, because the row is already on `main`. Nothing is owed to this reconcile for #1217. The reason the row's gate is not recorded as a pass on `main` is restated as `## Now` gives it, which is that the 96/96 was measured on the #1221 branch tree and no run against current `main` exists. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…as behind it (#1074) `scripts/agent-preflight.sh --fail-on-skip` SKIPPED both `commit-trailers` and `commit-style`, because `origin/main` was not an ancestor of this head. A SKIP reports nothing about this tree, so neither gate had read it. This merge is what lets them run. `origin/main` is `04f1cead6f94c01f8fd4e669148f7a9dbcf0b132`. The five commits are a container-publish repair (#1560), a BPE row close (#1558), two README edits (#1554, #1302), and a clock-gate spec (#1553). They touch 19 files, and three of them are files this pull request also edits: `.agents/issue-index.md`, `docs/BENCHMARKS.md` and `docs/STATUS.md`. `git merge-tree --write-tree` reported no conflict on any of the three, and the append-only and public-document gates are rerun on the merge result rather than assumed from that report. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

#1074) `origin/main` moved to `947e5f648552192d08cad4e636953be57bf34ffb` while this branch's gates were running, so the trailer gates would have SKIPPED again on the next run. A SKIP reports nothing about this tree. The one commit is the MiniMax-Music3 DiT profile (#1550). It touches 11 files, and two are files this pull request also edits: `.agents/issue-index.md` and `docs/BENCHMARKS.md`. Both auto-merged, and the append-only and public-document gates are rerun on the merge result rather than assumed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

mudler added 8 commits August 16, 2026 23:14

localai-bot changed the title ~~record(MODEL-TEXT-nemotron-h): the row's blocker landed 23 commits ago, and a file glob is why nobody saw it~~ record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it Aug 18, 2026

localai-bot mentioned this pull request Aug 18, 2026

Mamba2 SSD core is unported: KERNEL-SSM-MAMBA has only the GDN arm, so every hybrid-SSM arch (Nemotron-H/3.5, Bamba, Granite-4, Zamba2, Falcon-H1, Mamba2) is blocked #496

Open

mudler added 2 commits August 18, 2026 16:38

This was referenced Aug 18, 2026

measure(#810 A3): the token gate ran on real weights and FAILED 6/96, and the failure names the mechanism #1158

Closed

main is RED on build-test-cpu (test_qwen27_dense_forward), and main's own CI never reports because every run is CANCELLED #1285

Open

mudler added 5 commits August 19, 2026 11:32

localai-bot mentioned this pull request Aug 19, 2026

test_qwen3_5_decode_graph_seam SEGFAULTs in build-test-cpu, and build-test-cpu is red on every open PR #1422

Closed

localai-bot force-pushed the row/RECORDS-NEMOTRON-H-MATRIX-1074 branch from f30c2da to c598f1b Compare August 20, 2026 04:05

mudler added 2 commits August 20, 2026 07:55

localai-bot mentioned this pull request Aug 21, 2026

check-public-doc-tables.py does not test key uniqueness, so a keyed projection can carry two contradictory rows for one key #1543

Open

mudler added 4 commits August 21, 2026 07:59

Navigation Menu

record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it - #1081

record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it#1081
localai-bot wants to merge 24 commits into
mainfrom
row/RECORDS-NEMOTRON-H-MATRIX-1074

localai-bot commented Aug 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why a directory listing hid the blocker

The row STAYS INVENTORIED

The A3 gate PASSES, and it belongs to a named tree

The stale ## Owed bullet for #1217

Two defects found and fixed in flow

Which projections this change owes, and why

.agents/issue-index.md

Gates

Inherited, not introduced

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

localai-bot commented Aug 16, 2026 •

edited

Loading

The row STAYS `INVENTORIED`

The stale `## Owed` bullet for #1217

`.agents/issue-index.md`